The present invention relates generally to data modeling and in particular to modeling methods and systems using Gaussian mixtures.
Gaussian mixture is commonly used in parametric estimation of density functions and in unsupervised clustering. While the batch learning process of these models on stationary distribution is well understood and can be solved using the EM (expectation minimization) algorithm, there are increasing interests in developing online learning algorithms on dynamic data. The demand for such algorithms comes from real-time applications like video processing where a stream of new data is constantly being observed and the underlying data distribution may change over time.
Current solutions reported in the literature use adaptive filter learning to track slow distribution shifts, and handle sudden distribution changes through Gaussian reassignments. Ever since the original proposal of using Gaussian mixtures for modeling pixel distributions in video signals, the conventional approaches have followed the formulation presented by Stauffer, C. and Grimson, W. E. L., Adaptive Background Mixture Models for Real-time Tracking, Proc. CVPR, Vol. 2, pp 246–252, June 1999. At each step, parameters of one (or more, depending on the selection criteria) of the Gaussians that best match the new observation x are updated using a recursive filter θ(t)=(1−α)·θ(t−1)+α·{circumflex over (θ)}(x;t), where α controls the temporal rate of adaptation (also referred to as the “learning factor”, α). If x does not match the current model well, then one of the Gaussians is reassigned to the new point. Through recursive filter learning and Gaussian reassignment, the system is able to model dynamic distributions. Unfortunately, convergence is very slow using this strategy, requiring the distribution to remain stationary for a long time to allow the system to achieve an acceptable approximation.
While recursive filter learning is necessary to track distribution changes, a more efficient strategy can be used to speed up convergence during initial parameter estimation. Such a strategy was proposed by Kaew TraKulPong, P. and Bowden, R., An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection, Proc. of 2nd European Workshop on Advanced Video Based Surveillance Systems, September 2001. Kaew et al. proposed separating the learning process into discrete stages. By storing sufficient statistics of the first L samples in the early learning stage and applying the appropriate term weighting, convergence can be improved. However, this explicit division of learning stages can only be applied at initialization. It has been observed that, in fact, subsequent Gaussian reassignment also suffers from slow convergence. In addition, a buffer is needed to store statistics of the previous L samples.
The requirement of stationarity of the distribution of data is at odds with the dynamic nature of a real time data. Data modeling of real time data such as audio streams and video requires an improvement on conventional data modeling techniques.